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SUMMARY 


The  Independent  Validation  and  Verification  (IV&V)  contract  for  the  Advanced  Planning 
System  (APS)  program  represented  an  effort  to  explore  alternative  concepts  for  performance  of 
IV&V  in  the  Evolutionary  Acquisition/Rapid  Prototyping  (EA/RP)  environment.  The  EA/RP 
environment  with  its  continually  evolving  requirements  baseline  and  design,  is  fundamentally 
unsuited  for  the  application  of  traditional  IV&V  concepts  where  the  approach  is  predicated  on 
one  phase  of  the  development  effort  completing  prior  to  commencing  the  next  phase. 


Command  and  Control  (C^)  software  has  always  been  difficult  if  not  impossible  to  develop  on 
time  and  on  budget.  The  requirements  are  dynamic,  and  the  very  large  software  systems  needed 
take  either  long  development  cycles  with  a  small  team,  or  long  development  cycles  with  a  large 
team.  The  only  difference  being  in  the  amount  of  coordination  and  communication  required. 
The  EA  approach  was  adopted  by  the  Joint  Logistics  Commander's  Panel  as  a  solution  to  this 

problem.  The  APS  was  one  of  the  first  C^  systems  to  implement  this  acquisition  approach. 

Specific  solutions  to  the  dynamic  nature  of  the  IV&V  problem  were  to  apply  three  tactics  in 
employing  the  IV&V  resources.  Rather  than  attempt  100%  coverage  of  product  for  each 
prototype  released,  spot  samples  were  chosen  based  on  critical  functional  factors.  In  order  for  a 
sampling  theory  to  succeed,  a  new  perspective  of  analysis  was  derived  to  review  the  software 
from  a  vertical  perspective,  i.e.,  analyze  the  requirements,  design,  and  implementation  of  a 
particular  function.  To  ensure  the  highest  productivity,  and  reduce  schedule  time,  maximum  use 
of  Computer  Aided  Software  Engineering  (CASE)  tools  were  employed. 


The  implementation  of  this  strategy  allowed  the  IV&V  contractor  to  focus  the  available 
resources  on  those  areas  that  were  most  crucial  to  the  success  of  the  program  at  a  given  point  in 
time.  During  the  prototyping  phase,  the  IV&V  efforts  were  concentrated  on  new  and  evolving 
functionality.  This  approach  allowed  the  IV&V  contractor  to  maximize  the  effectiveness  of 
limited  resources  and  still  provide  comprehensive  coverage  of  the  APS  product. 
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INTRODUCTION 


APPLICATION  SYSTEM  DESCRIPTION 

The  APS  is  an  integrated,  force  level  air  battle  planning  system  which  supports  personnel  within 
the  force  level  structure  of  the  Theater  Air  Control  System  (TACS)  to  develop  air  battle  plans 
and  generate  air  tasking  to  achieve  the  objectives  of  the  Joint  Forces  Air  Component 
Commander's  (JFACC)  overall  strategy.  The  force  level  element  of  the  TACS  is  the  Air 
Operations  Center  which  is  used  by  the  JFACC  to  manage  the  application  of  combat  air  power. 
The  APS  provides  the  JFACC  with  the  means  to  organize,  plan,  direct  and  coordinate  organic 
aviation  elements  with  those  of  other  services  and  countries  during  joint  or  combined  combat 
operations.  The  APS  provides  the  JFACC  with  enhanced  capabilities  which  enable  the  Combat 
Plans  Division  (CPD)  to  perform  the  following  tasks  more  effectively: 

a.  Coordinate  air  operations  with  higher,  adjacent  and  support  headquarters. 

b.  Coordinate  apportionment  recommendation  meetings. 

c.  Determine  allocations. 

d.  Perform  air  defense  and  tactical  air  support  planning. 

e.  Establish  and  maintain  planned  sortie  rates. 

f.  Conduct  mission  planning. 

g.  Prepare  the  Air  Tasking  Order  for  distribution. 

h.  Receive,  store  and  utilize  intelligence  information. 

The  APS  is  one  of  the  first  software  only  systems  to  obtain  a  formal  system  nomenclature  as  the 
AN/TYQ-43  V(l.O). 

PROBLEM  DOMAIN 

The  Command  and  Control  (C^)  mission  area  has  always  been  characterized  by  the  inability  to 
define  in  detail  the  operational  capabilities  and  functional  characteristics  of  a  system  before  the 
1  2 

start  of  development.  ’  To  solve  this  paradox,  the  concept  of  evolutionary  acquisition  was 
introduced  for  the  development  of  C^  systems.  Evolutionary  Acquisition  (EA)  is  defined  as: 

Evolutionary  acquisition  is  an  acquisition  strategy  which  may  be  used  to  procure  a  system 
expected  to  evolve  during  development  within  an  approved  architectural  framework  to  achieve 
an  overall  system  capability.  An  underlying  factor  in  evolutionary  is  the  need  to  field  a  well- 
defined  core  capability  quickly  in  response  to  a  validated  requirement,  while  planning  through 
an  incremental  upgrade  program  to  eventually  enhance  the  system  to  provide  the  overall  system 
capability.  These  increments  are  treated  as  individual  acquisitions,  with  their  scope  and  content 


^  "Report  of  the  Defense  Science  Board  Task  Force  on  Command  and  Control  Systems  Management,"  July  1978,  Office  of  the  Under  Secretary 
of  Defense  Research  and  Engineering,  Washington,  D.C. 

2  o 

"Command  and  Control  (C*^)  Systems  Acquisition  Study  Final  Report."  September  1, 1982,  The  Armed  Forces  Communications  and  Electronic 
Association,  Falls  Church,  Va. 
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being  the  result  of  both  continuous  feedback  from  developing  and  independent  testing  agencies 
and  the  user  (operating  forces),  supporting  organizations,  and  the  desired  application  of  new 

technology  balanced  against  the  constraints  of  time,  requirements,  and  cost.^ 

EA  did  not  offer  a  complete  solution  for  the  development  and  fielding  of  the  APS.  The 
definition  of  the  fieldable  core,  was  too  complex  to  be  specified  at  the  outset  of  the  program, 
therefore  the  APS  Program  Office  applied  the  Rapid  Prototyping  (RP)  development  approach 
during  the  core  system  evolution  phase.  A  series  of  prototypes  was  developed  to  gain  user 
involvement  and  feedback.  The  system  core  functionality  was  prototyped  in  a  series  of 
increments.  Each  increment  demonstrated  the  maturation  of  previous  capabilities  while 
providing  additional  capability  for  user  review  and  comment.  The  integration  of  EA  and  RP 
methodologies  was  a  very  effective  approach  for  developing  the  system.  However,  in  this 
dynamic  and  fast  paced  development  environment,  the  usual  methods  of  performing  IV&V  were 
no  longer  germane. 

IV&V  is  best  described  as  an  evaluation  "by  an  independent  contractor,  to  insure  that  the 
software  properly  performs  all  intended  functions  and  of  equal  importance  that  it  performs  no 

unintended  functions"^.  In  order  to  effectively  evaluate  a  system,  IV&V  practices  should 
"evaluate  software  through  all  phases  of  the  life  cycle:  requirement  definition,  design,  coding, 
test  and  integration.  Adherence  to  standards,  schedules,  design,  procedures,  etc.  (should)  also 
(be)  evaluated". 

The  IV&V  effort  for  APS  was  designed  to  provide  a  comprehensive  evaluation  during  APS 
development  phases  III,  IV,  and  V  of  the  software  project  to  ensure  that: 

1) Errors  are  detected  and  corrected  as  early  as  possible  in  the  software  life  cycle; 

2) Project  risk,  cost  and  schedule  effects  are  lessened; 

3) Software  quaUty  and  reliability  are  enhanced; 

4) Management  visibility  into  software  process  is  improved; 

5) Proposed  changes  and  their  consequences  can  be  quickly  assessed. 


The  IV&V  requirements  had  to  be  satisfied  even  though  the  APS  program  was  tasked  to  build 
and  field  a  system  more  rapidly  than  the  typical  acquisition  cycle  provides.  The  methodology 
chosen  for  the  development  effort  was  the  EA/RP  methodology.  This  methodology  was  based 
upon  a  top  level  set  of  system  requirements  being  defined  or  hypothesized,  which  are  then 
continually  refined  in  an  iterative  manner  through  a  series  of  incremental  software  prototypes 


*  “Evolutionary  Acquisition:  An  Alternative  Strategy  for  Acquiring  Command  and  Control  (C^)  Systems,"  March  1987,  Ihe  Defense  Systems 
Management  College,  Fort  Belvoir,  VA 

^  USAF  Space  Division  "Management  Guide  for  IV&V"  August  1980 
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that  let  the  user  community  actually  use  the  system  and  provide  feedback  to  the  developer  during 
the  development  cycle.  The  EA/RP  approach  represents  a  marked  departure  from  the  classical 
systems  engineering  or  traditionar‘waterfaH”  development  model.  (See  Figure  1.)  The  major 
departure  involved  with  the  EA/RP  methodology,  is  that  the  system  and  its  requirements  evolve 
during  development,  vice  deriving  and  defining  the  requirements  as  a  separate  activity  presaging 

the  rest  of  the  development^  as  is  the  norm  with  a  traditional  development  effort.  The  EA/RP 
development  model  more  closely  follows  the  spiral  development  model  of  Dr.  Barry  Boehm. 

(See  Figure  2.) 

The  result  of  this  EA/RP  approach  on  APS,  is  that  the  detailed  requirements  and  design  actually 
developed  at  the  same  pace  as  the  deliverable  software.  This  EA/RP  approach  therefore  dictated 
that  alternative  technologies  and/or  methodologies  for  the  performance  of  the  IV&V  effort  must 
be  developed  as  the  classical  IV&V  techniques/methodologies  are  associated  with  a  development 
effort  following  the  traditional  model. 

The  IV&V  role  is  typically  concerned  with  answering  the  following  two  questions  “Are  we 
building  the  product  right?”  (Verification),  and  “Are  we  building  the  right  product?” 

9 

(Validation)  .  Typically,  much  of  the  effort  is  spent  reviewing  the  requirements  for 
completeness,  consistency,  testability,  etc..  Then  the  requirements  are  compared  with  the  top 
level  and  detailed  designs.  As  the  code  comes  into  being,  it  is  then  compared  to  the  design  and 
the  requirements.  After  the  evaluation,  the  code  is  tested  against  the  requirements  and 
engineering  design.  The  labor  and  manpower  required  to  perform  this  task  on  the  APS 
following  accepted  IV&V  methodology  was  determined  to  exceed  the  cost  of  the  system 
development. 

As  a  result  of  this  analysis,  three  approaches  were  selected  to  solve  this  dilemma: 

1)  Focus  on  critical  areas  of  the  code,  and  apply  sampling  theory  to  verify  the  product. 
This  concept  became  known  as  “Spotlighting”  to  the  APS  development  team,  as  it  provided  a 
means  to  focus  the  available  resources  on  critical  areas. 

2)  Explore  the  application  of  Computer  Aided  Software  Engineering  (CASE)  tools  and 
usage  of  a  modern  Software  Engineering  Environment  (SEE).  The  application  of  reverse 
engineering  technology  offered  leverage  for  both  the  development  team  and  IV&V  contractor  to 
improve  productivity. 


3)  Concentrate  analysis  activities  on  functional  problems.  This  concept  was  termed 
"Veracity"  because  the  goal  was  to  determine  the  truthfulness,  completeness,  accuracy, 
precision,  fidelity,  and  integrity  of  an  algorithm  across  the  executing  units  and  databases  utilized. 


^Hawks,  Kenneth  B,  New  Methodologies  foE  Verification  Md  Validation  ia  Evolutionary  Acquisition,  AIAA-91-3820-CP,  p,  780. 
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Barry,  Boehm,  Software  Engineering  Economics.  Prentice-Hall,  1981,  p.37. 
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Figure  2  -  Spiral  Development  Model  ^ 


^  "A  Spiral  Model  of  Software  Development  and  Enhancement”,  May  1988,  IEEE  COMPUTER,  IEEE  Headquarters,  345  E.  47th  Street,  New 
York,  NY  10017. 
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OVERALL  IV&V  STRATEGY 


Technical  Approach 

The  APS  IV&V  problem  is  multi-dimensioned,  and  extremely  complex.  There  were  numerous 
unique  aspects  of  the  APS  IV&V  concept.  Due  to  the  dynamic  nature  of  the  system 
development  schedule,  several  factors  influenced  the  IV&V  approach.  Timeliness  was  critical. 
Finding  errors  and  getting  them  fixed  could  not  be  a  six  month  long  process,  when  a  new 
prototype  was  being  introduced  at  six  month  intervals.  A  balance  between  and  among  resources 
was  paramount  since  not  every  function  point  could  be  evaluated  in-depth  for  each  release.  This 
led  to  one  of  the  primary  concepts  of  the  APS  IV&V,  which  was  the  emphasis  of  analyzing 
critical  areas,  vice  a  comprehensive  review  of  all  requirements  implementation.  Once  chosen  for 
evaluation,  an  area  under  development  was  examined  in  depth. 

The  IV&V  team  was  small,  thus  skill  application  and  efficient  use  of  the  individual  skills  was 
very  important.  When  an  error  was  found  by  either  the  software  development  or  CASE  tools, 
the  first  task  was  to  validate  that  the  answer  supplied  by  the  tool  was  correct.  In  a  similar 
manner.  Commercial-off-the-shelf,  and  "netware"  integrated  into  the  program  was  also  evaluated 
for  errors.  In  every  activity,  the  contribution  to  the  APS  system  also  had  to  kept  in  mind.  An 
important  element  was  to  identify  the  source  and  type  of  error.  Errors  could  be  caused  by  faulty 
design;  faulty  algorithmic  implementation;  faulty  database  design,  consistency,  or  erroneous 
data;  or  inappropriate  operator  actions.  Likewise  the  source  of  errors  could  also  be  a  result  of 
problems  in  the  documentation,  designs,  implementation,  code,  or  usage.  (See  Figure  3.) 


The  fundamental  strategy  was  to  Evaluate  -  Assess  -  Recommend.  This  paradigm  was  the  key 
element  in  the  accomplishment  of  all  tasks.  The  evaluation  was  tailored  to  the  particular 
function  point  being  reviewed.  The  assessment  was  scoped  to  fit  the  desired  end  goal  of  depth 
and  detail.  One  of  the  cardinal  ground  rules  was  that  every  evaluation  have  a  recommended 
solution.  In  this  manner  the  evaluation  process  contributed  to  the  development  process  by 
suggesting  better  solutions  as  well  as  finding  weakness  in  the  developing  product.  All  to  often 
rV&V  efforts  result  in  the  IV&V  team  just  finding  problems  without  having  the  responsibility  to 
assess  the  problem  and  propose  a  solution.  This  results  in  an  adversarial  relationship  between 
developer  and  IV&V  teams,  and  in  extreme  inefficiency  for  the  development  team,  as  effort  has 
to  be  wasted  to  validate  the  problem,  measure  its  scope,  figure  out  a  solution,  and  finally 
implement  the  solution.  While  the  development  team  is  not  obligated  to  implement  the  solution 
proposed  by  the  IV&V  contractor,  the  evaluate  -  asses  -  recommend  approach  assured  that  the 
entire  problem  was  understood,  and  that  a  solution  did  exist. 
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Fiqure  3  -  APS  FV&V  Model  is  multi-dimensioned 
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The  Verification  and  Validation  areas  were  established  as  follows  to  support  the  new  IV&V 
paradigm  in  the  EA/RP  environment. 

Verification  was  formally  scoped  as  five  task  areas: 

Testing  oversight  and  conduct. 

Database  content  and  review. 

Interface  review. 

Specification  review. 

Audit  support  for  Physical  Configuration  Audit. 

While  these  task  assignments  are  similar  to  the  traditional  IV&V  efforts,  how  they  were 
executed  in  the  EA/RP  environment  was  radically  different.  Testing  oversight,  database  content, 
interface  review,  and  specification  review  were  performed  on  a  function  point  basis  using 
sampling  techniques.  As  a  functional  evaluation  was  performed,  the  requirements  through 
implementation  documents  were  reviewed,  and  the  quality  of  the  function  measured.  The 
reverse  engineering  capabilities  of  the  SEE  tool  set  provided  the  capability  to  audit  the  C  and 
Ada  code  for  PCA  almost  entirely  via  computer. 


Validation  was  formally  scoped  as  five  tasks: 

Formal  Qualification  Testing  -  User  Evaluations. 
Requirements  Satisfaction. 

Concepts  of  Operations  Implementation 
User  Expectations 

Audit  support  for  Functional  Configuration  Audit. 


Validation  was  performed  by  the  user  community  until  the  acceptance  of  the  common  core 
release  of  the  APS.  The  IV&V  contractor  performed  a  vital  role  between  the  development  team 
and  user  by  recording  and  analyzing  the  comments  received.  Upon  fielding  of  the  system  core, 
the  IV&V  contractor  became  prime  for  the  Validation  tasks 

Management 

To  address  the  needs  of  performing  IV&V  in  an  EA/RP  environment,  there  were  certain 
limitations/constraints  that  needed  to  be  acknowledged  and  appropriate  measures  defined  to  deal 
with  them.  The  primary  limitation(s)  were  the  prototyping  schedule  and  the  continually 
evolving  requirements  baseline  as  the  prototypes  incorporated  increasing  amounts  of 
functionality.  The  typical  IV&V  methodology  which  is  designed  to  support  a  classical 
development  program  (i.e.  Boehms’  waterfall  model)  were  clearly  inappropriate  in  the  context 
of  the  EA/RP  environment. 

To  respond  to  these  issues  a  specific  blend  of  personnel  were  defined  and  the  concepts  of 
“Spotlighting”  and  “Veracity”  were  evolved  to  guide  the  staff  in  performance  of  the  IV&V  task. 
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The  contract  itself  was  structured  to  allow  control  of  the  IV&V  effort,  with  a  maximum  of 
flexibility  for  the  contractor.  Overall,  three  levels  of  reporting  structure  were  implemented.  At 
the  top  was  the  program  level  IV&V  Program  Plan.  This  plan  outlined  the  resources  available, 
the  management  structure  of  the  IV&V  team,  and  the  broad  guidance  for  the  approach.  The 
Program  Plan  was  revised  quarterly  to  reflect  near  term  goals  and  objectives,  and  to  identify 
specific  resources  to  be  applied  to  specific  areas,  such  as  testing  or  auditing.  The  monthly  status 
reports  provided  a  means  to  fine  tune  the  application  of  resources,  and  communicate  the  status  of 
tasks  in  process. 

Feedback  loops  for  the  IV&V  process  were  provided  in  the  technical  problem  reporting  system, 
trip  and  technical  reporting,  and  in  appendixes  to  the  monthly  reports.  As  technical  problems 
were  revealed,  an  evaluation  report  was  rendered  electronically  and  provided  to  the  APS 
program  office.  The  program  office  then  reviewed  the  report  and  electronically  forwarded  it  to 
the  contract  team  for  action  as  appropriate.  As  individual  areas  were  investigated,  technical 
reports  were  rendered  electronically  immediately.  This  approach  assured  that  overall  findings 
could  be  reviewed  and  forwarded  to  the  contract  team  in  a  timely  manner.  If  travel  was  required 
to  review  a  developing  function,  the  overall  report  of  the  investigation  was  contained  in  the  trip 
report.  Several  tasks  were  long  term  in  nature,  i.e.,  oversight  of  the  configuration  management 
system,  and  these  tasks  were  statused  in  the  monthly  reports.  Specific  problems  found  in  long 
term  tasks  were  documented  in  technical  reports  or  evaluation  reports  as  applicable. 

This  management  framework  provided  the  visibility  for  the  APS  Program  Office  to  stay  abreast 
of  and  control  the  IV&V  team,  while  also  getting  the  output  of  the  IV&V  tasks  in  a  timely 
manner.  This  approach  minimized  the  cost  of  report  preparation,  maximized  customer  visibility, 
and  ensured  timely  reporting  of  problem  areas. 

Personnel 

Reflecting  the  dynamics  of  the  EA/RP  environment,  a  crucial  component  to  the  success  of  the 
program  was  the  composition  of  the  IV&V  staff.  The  APS  IV&V  effort  required  a  small  group 
of  engineers  with  a  diverse  background  to  successfully  execute  the  contract  as  envisioned.  The 
personnel  required  a  combination  of  skills  that  typically  would  only  be  found  in  a  larger 
organization;  the  following  areas  of  expertise  were  defined  at  the  outset  of  the  contract  as  the 
mandatory  skill  set  for  the  team  to  possess  and  served  as  the  basis  for  all  staffing  decisions 
throughout  the  entire  period  of  performance  The  skills/areas  of  expertise  were: 

Extensive  background  in  Command,  Control,  Communications  (C3)  decision  aids; 

Software  Configuration  Management; 

Experience  in  the  following  programming  languages:  C,  SQL,  and  Ada; 

Software/system  level  testing  experience; 

Experience  in  the  use  of  modern  Software  Engineering  Environments; 

Experienced  data  base  designer/data  analyst,  and; 

Experienced  system  engineer. 

CTA  management  was  extremely  successful  in  finding  and  retaining  staff  with  the  appropriate 
qualifications  throughout  the  life  of  the  contract.  Key  personnel  remained  with  the  contract  from 
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the  initial  startup  through  the  eventual  lamp  down  after  the  decision  was  made  to  not  extend  the 
period  of  performance  as  planned.  Without  the  ability  to  attract  and  retain  personnel  for  the 
length  of  the  program,  the  knowledge  base  and  expertise  required  to  rapidly  assess  changes  and 
their  impact  upon  APS  would  have  significantly  diminished  the  ability  to  execute  the  IV&V 
concept  as  envisioned  by  the  Government  and  CTA. 
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IMPLEMENTATION 


A  series  of  technical  exchange  meetings  were  held  between  the  program  office  team  and  the 
rV&V  team  to  establish  how  the  IV&V  strategy  would  be  applied  in  detail.  This  "technical 
action  group"  defined  how  the  concepts  of  "veracity”,  “spotlighting",  and  the  CASE  tools  would 
be  applied. 


Veracity 

The  veracity  concept  was  constructed  to  ensure  that  the  fundamental  building  blocks  of  the 
planning  system  reflected  reality  and  were  accurate.  The  APS  IV&V  was  implemented  using  a 
sampling  technique  whereby  key  elements  of  the  system  were  subjected  to  a  thorough  and  in- 
depth  review  from  a  vertical  perspective  vice  the  classical  horizontal  perspective.  In  a 
traditional  program,  the  requirements  baseline  would  be  established  and  the  IV&V  contractor 
would  perform  a  thorough  review  of  that  baseline.  Next  a  design  would  be  developed  to 
implement  the  requirements,  and  finally  code  would  be  produced  which  implemented  the  design. 
This  review  process  essentially  looks  at  the  system  horizontally  over  time.  At  each  stage,  the 
review  process  goes  back  only  to  the  prior  stage,  the  entire  system  is  not  examined  at  once.  By 
adopting  a  vertical  perspective  and  implementing  sampling  technique,  key  portions  of  the  system 
are  reviewed  in  their  entirety  (requirements,  design,  implementation)  at  once.  Based  upon  the 
results  of  the  various  samples,  problem  areas  can  be  identified  quickly  and  specific  problem 
areas  can  be  tracked/managed  via  trend  analysis. 

Veracity  is  both  an  analysis  perspective  and  an  analysis  criteria.  It  can  be  the  framework  for 
gaining  a  perspective  on  an  algorithm  or  contiguous  component  of  the  software.  The  dictionary 
1  2 

and  thesaurus  lists  the  following  as  components  of  veracity: 

Truthfulness  -  of  logic. 

Accuracy  -  of  calculation,  of  results. 

Precision  -  of  data,  results. 

Faithfulness  -  of  implementation  of  requirements;  design. 

Fidelity  -  accuracy  of  description. 

Integrity  -  cohesiveness  with  related  algorithms,  autonomy  of  self. 

Thus  when  a  function  is  viewed  from  the  standpoint  of  veracity,  a  requirement  is  reviewed, 
compared  to  design,  the  design  compared  to  the  code,  testing  results  evaluated,  and  the 
interaction  of  the  function  with  related  functions  are  also  reviewed.  By  reviewing  the  various 
levels  of  abstraction  of  the  software,  a  perspective  of  implementation  of  a  particular  algorithm  is 
gained  and  a  quality  assessment  made. 

The  veracity  concept  worked  extremely  well  in  the  APS  environment  where  the  essential 
structures  of  the  system  models  could  be  captured  and  analyzed  via  the  spotlight  technique.  In 


’’Webster's  New  World  Dictionary",  Third  College  Edition,  Simon  &  Schuster,  1988 
"Roget’s  College  Thesaurus",  43rd  Edition,  Signet  Books,  1962 
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this  case  the  fundamental  underpinning  a  of  the  system  models  were  documented  in  the  database 
structure,  before  code  had  been  developed  to  access/manipulate  the  database  tables.  As  a  case  in 
point ,  the  targeting  model  and  the  various  relationships  were  defmed/refmed  extensively  prior 
to  any  code  being  written  to  implement  the  actual  target  planning  logic. 

Once  selected  for  evaluation,  the  depth  and  detail  of  verification  was  established.  Evaluation 
criteria  consisted  of  data  consistency  across  all  units  of  the  program,  completeness  of  the  models 
that  were  being  developed  and  then  recording  these  results  both  on  a  point  in  time  sample  as 
well  as  tracking  trends  in  a  particular  compilation  unit  over  time.  The  concept  of  veracity 
allowed  IV&V  personnel  the  latitude  to  track  the  interactions  of  a  function  among  and  within 
related  functions,  while  providing  boundary  constraints. 

Spotlighting 

Spotlighting  is  an  approach  to  accomplishing  IV&V  using  sampling  techniques.  In  the  context  of 
APS  it  can  be  though  of  as  shining  a  strobe  light  into  a  particular  area  of  concern  at  some  point 
in  the  program  and  evaluating  the  completeness  of  the  design,  the  accuracy  of  the 
implementation,  etc..  Samples  were  based  upon  the  criticality  of  new  functionality,  volume  of 
user  feedback,  complexity  of  the  function,  and  gut  feel  of  the  program  office  personnel.  Since 
functionality  was  either  being  developed  from  scratch  or  being  constantly  refined,  any 
evaluation  was  good  for  only  a  very  brief  period  of  time.  Thus  the  perception  of  the  function 
being  in  the  spotlight  for  a  few  moments,  then  fading  into  the  background. 

The  selections  picked  for  sampling,  were  chosen  by  conventional  means.  New  functionality  that 
the  users  felt  was  important;  previous  functionality  up  dated  because  of  user  feedback;  and 
functions  not  normally  visible  to  the  user,  but  critical  to  the  performance  of  the  system. 

Spotlighting  allowed  the  IV&V  Program  Manager  the  flexibility  to  coordinate  the  IV&V  staff 
efforts  with  the  Government  schedule  and  focus  the  available  resources  on  those  areas  that  were 
crucial  to  the  success  of  the  program.  Spotlighting  can  also  be  applied  as  a  “tiger  team”  concept 
where  the  intent  was  to  go  in,  thoroughly  examine  a  particular  area  and  then  move  on  to  the  next 
area  deemed  crucial.  There  were  times  when  this  ability  to  "hit  an'  git"  proved  very  useful  for 
committing  to  program  milestones. 

CASE  Tool  Application 

A  necessary  component  of  the  IV&V  methodology  was  to  leverage  the  investment  and  usage  of 
automated  tools  to  aid  both  the  development  and  IV&V  teams.  This  was  particularly  important 
given  the  small  number  of  personnel  assigned  to  the  program  and  the  requirement  for  a  complete 
new  release  every  three  to  six  months  during  the  prototyping  phase  of  the  program.  A  number 
of  SEE’s  were  investigated  which  were  available  in  the  late  1990  time  frame.  The  requirements 
were  that  the  SEE  chosen  had  to  be  available  on  both  the  DECstation  and  Sun  SPARCstation 
environment,  and  provide  tools  that  would  provide  assistance  in  the  design,  test,  documentation, 
and  configuration  management  areas.  A  critical  requirement  was  the  ability  to  automatically 
support  reverse  engineering  of  existing  code. 
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The  program  did  not  have  the  schedule  time  nor  manpower  to  correlate  the  documentation  with 
the  actual  code.  A  significant  amount  of  code  was  acquired  from  existing  sources.  Internal 
Research  and  Development  (IR&D),  public  domain  software  from  the  Internet,  commercial 
software,  legacy  and  existing  software,  all  were  incorporated  into  the  APS.  The  SEE  tools  also 
had  to  increase  engineer  productivity,  and  result  in  a  product  that  was  of  higher  quality  and 
reliability,  and  maintainable  when  turned  over  to  the  Air  Eorce  at  the  conclusion  of  the  program. 

The  IV&V  contractor  evaluated  a  number  of  CASE  tools  and  recommended  that  a  modern  SEE 
based  on  the  Cadre  Teamwork  CASE  tool  be  adopted  by  the  APS  program  for  usage  both  by  the 
developer  and  the  IV&V  staff.  This  CASE  tool  allowed  maximum  leverage  of  the  available 
personnel  by  supporting  a  variety  of  program  activities  (design,  coding,  testing,  analysis,  and 
document  generation)  within  a  single  environment  that  can  be  shared  by  all  personnel.  The 
Cadre  product  provided  extensive  design  and  coding  support  as  well  as  a  reverse  engineering 
tool  that  allowed  structure  charts/graphs  to  be  constructed  for  a  given  release  for  both  the  Ada 
and  C  code.  It  also  provided  the  ability  to  generate  the  DoD  standard  2167A  documentation  that 
was  required  for  APS.  The  reverse  engineering  capability  provide  a  quick  view  into  what  is, 
while  the  requirements  and  design  documents  provided  a  view  what  was  intended.  A  quick 
comparison  of  the  two  revealed  deviations  that  could  be  evaluated. 

The  combination  of  these  techniques/methodologies  allowed  the  IV&V  team  to  provide  real  time 
feedback  to  the  Government  with  respect  to  the  quality  and  completeness  of  the  APS  system  as  it 
was  evolving.  Without  these  techniques  and  methodologies,  CTA  would  have  been  analyzing 
requirements,  design,  and  code  baselines  that  were  significantly  out  of  date  with  respect  to  the 
current  baseline  and  would  not  have  been  as  responsive  as  required  to  support  the  dynamics  of 
an  EA/RP  environment. 

CodeCenter  from  Centerline  software  provided  extensive  testing  capabilities  for  the  portion  of 
APS  developed  in  C.  In  the  language  context,  it  provided  similar  checking  to  the  capabilities  of 
an  Ada  compiler.  It  provided  a  useful  tool  in  the  context  of  FCA/PC A  to  examine  the  code  for 
compliance  with  the  appropriate  coding  standards  and  correctness  automatically.  The  audit  of 
the  code  to  its  documentation,  was  also  done  automatically.  The  use  of  the  SEE  made  it  possible 
to  perform  a  FCA  and  PCA  with  each  major  release  of  the  APS. 

The  adoption  of  these  products  represented  a  significant  cost  to  the  Government  as  the 
Government  acquired  the  required  licenses  for  both  IV&V/Government  use  and  the  development 
team.  This  cost  was  off-set  by  a  higher  quality  product  and  increased  productivity.  However, 
the  results  that  were  achieved  using  the  tools  did  not  realize  the  full  potential  of  the  tool  suite. 
This  was  due  to  a  number  of  factors,  the  most  significant  one  being  that  the  tools  were  not  used 
in  the  forward  direction,  i.e.,  during  the  design  phase  of  a  particular  function,  thus  the  capability 
of  the  tools  to  evaluate  the  designs  prior  to  coding  was  lost.  In  subsequent  development  efforts, 
a  SEE,  such  as  that  selected  for  APS,  should  be  defined  prior  to  the  start  of  the  detailed  design 
process,  the  entire  development  team  needs  to  have  access  to  it,  and  a  commitment  must  be  in 
place  to  use  it  to  support  the  entire  development  process. 
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Configuration  Management/Change  Tracking  in  EA/RP  Environment 


One  of  the  key  tasks  in  a  EA/RP  environment  is  maintaining  a  working  control  of  the  various 
baselines,  in  particular  the  user  requirements  and  code  baselines.  In  point  of  fact,  given  the 
highly  iterative  nature  of  an  EA/RP  environment  controlling  the  requirements  baseline  is  even 
more  crucial  than  in  a  classical  development  environment  as  the  tendency  towards  requirement 
creep  is  heightened  due  to  the  constant  interaction  between  the  developers  and  the  users.  This 
environment  tends  to  enhance  the  natural  desire  to  provide  the  user  with  changes  and/or 
modifications  in  a  rapid  manner  without  always  taking  the  time  to  evaluate  the  overall  impact  on 
cost  and/or  schedule  constraints.  In  recognition  of  this  tendency,  the  Government  charged  the 
IV&V  contractor  with  the  responsibility  for  both  administering  the  Program  level  Change 
Management  and  Tracking  (CMAT)  component  of  the  APS  prototype  and  Evaluation  Reporting 
System  and  monitoring  and  assessing  the  developer's  Configuration  Management  Practices. 

This  approach  provided  multiple  benefits  to  the  APS  program.  First  and  foremost,  the 
Government  maintained  the  top  level  set  of  system  requirements  as  documented  in  CMAT 
throughout  the  life  of  the  program.  It  also  allowed  the  developer  to  focus  on  implementing  a  set 
of  Configuration  Management  practices  for  the  code  that  were  applicable  for  the  APS 
environment.  The  use  of  the  SEE  allowed  the  development  team  more  latitude  in  the  degree  of 
discipline  required  in  the  configuration  management  (CM)  philosophy  applied. 

User  Evaluations  often  generated  over  300  comments,  thus  the  APS  Program  Office  needed  a 
method  for  organizing  user  comments,  and  product  improvement  requests.  To  support  the 
volume  of  comments  generated  by  the  evaluators  a  reliable,  flexible,  and  user  friendly  tool  that 
would  allow  the  incorporation  of  this  information  into  a  single  automated  system  was  dictated. 
The  system  also  needed  to  incorporate  the  previous  unresolved  user  comments  into  the  "next 
phase"  of  APS  while  providing  identical  content  between  the  Program  Office  and  the  Software 
Development  Team. 

The  APS  Program  Office  also  realized  that  a  need  existed  for  a  system  that  would  allow  for  the 
recording  of  Software  Change  Requests  and  Preplanned  Product  Improvement  (P3I)  requests 
generated  during  the  Theater  Tailoring  portion  of  APS.  After  several  COTS  products  were 
reviewed  and  found  to  be  inadequate,  the  CMAT  system  was  developed  to  assist  the  APS 
Program  Office  and  the  Software  Development  Team  in  the  management  of  the  APS  user 
comments.  Software  Change  Requests  and  P3I  requests. 

During  the  EA/RP  phases  of  the  APS  contract,  the  APS  Program  Office  used  the  CMAT  system 
to  record  all  APS  comments  from  the  user  community.  The  user  comments  were  then 
disseminated  by  the  APS  Program  Office  and  those  deemed  appropriate  were  forwarded  to  the 
software  developers  for  inclusion  into  future  releases  of  APS.  The  APS  Program  Office 
required  a  system  that  would  be  flexible  enough  to  allow  recording  of  user  comments,  bug 
reports  and  future  enhancements,  while  providing  a  quick  turn  around  from  the  "users"  to  the 
developers.  When  APS  migrated  into  a  more  classical  software  development  effort  during  the 
fielding  and  maintenance  phases  where  stricter  configuration  management  practices  were 
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required,  CM  AT  was  used  to  record  SCRs,  corrective  software  actions  and  user  desired 
enhancements  (i.e.  P3I  items). 

APS  CMAT  system  consists  of  two  subsystems.  The  first  of  these  was  developed  by  CTA 
Incorporated  for  the  APS  Program  Office's  use  and  includes  the  programs,  forms,  tables  and 
procedures  required  by  the  Government  to  track  and  maintain  the  top  level  programmatic 
baseline.  The  second  CMAT  subsystem  was  developed  by  Unisys  for  use  by  their  development 
personnel. 

Within  CMAT  at  the  Program  Office,  SCRs  are  tracked  and  serve  as  input  to  CMAT  at  the 
developer's  site.  Within  CMAT  at  Unisys,  the  SCR  becomes  a  software  problem  report  (SPR) 
and  is  worked  by  the  development  team. 

As  the  CMAT  Administrator  CTA  also  generated  all  CMAT  reports.  These  reports  were 
required  for  all  informal  and  formal  reviews.  Formal  reviews  occurred  at  the  RL  Software 
Change  Review  Board  (RL  SCRB)  which  included  members  of  the  APS  Program  Office  and  the 
CTA  IV&V  team.  The  second  formal  review  occurred  at  the  APS  Software  Change  Review 
Board  (APS  SCRB)  which  included  members  of  the  APS  Program  Office,  the  developer's  team 
and  CTA  for  administration  and  support. 

The  IV&V  team  was  involved  in  the  review  of  the  open  SCRs  to  determine: 

(1)  The  scope  and  nature  of  the  SCR 

(2)  Possible  SCR  duplication 

(3)  Possible  consolidation  of  SCR  with  others 

(4)  Clarity  of  SCR 

(5)  What  is  the  SCR  assessment?  (IN  SCOPE,  OUT  OF  SCOPE) 

(6)  What  is  the  SCR  disposition?  (P3I,  CORE,  THEATER) 

(7)  What  is  the  appropriate  STATUS  for  the  SCR? 

(8)  Does  SCR  need  formal  review  by  the  RL  SCRB? 

SCRs  and  User  comments  were  reviewed,  subjected  to  formal  configuration  control  boards,  and 
upon  implementation  into  the  APS,  were  tested  for  closure.  When  a  new  version  of  the  APS 
was  released,  the  IV&V  team  would  verify  the  list  of  SCRs  reported  as  corrected  in  the 
developers  Version  Description  Document  (VDD)  against  the  CMAT  status.  CTA  then  provided 
the  SCR  list  to  the  test  team  members  (CTA  and  Government)  to  verify  that  the  change(s)  as 
implemented  corrected  the  problem  described.  This  was  done  using  the  SEE. 

The  process  employed  on  APS  provided  the  Government  and  the  IV&V  effort  valuable  insight 
and  a  high  degree  of  control  over  the  change  process  during  the  APS  development.  The  one  area 
where  significant  improvement  could  be  achieved  in  the  future  is  in  a  consolidated  Change 
Request/Problem  report  database  that  is  used  by  the  program  team  (Government,  IV&V,  and 
developer)  staff.  Significant  amounts  of  time  were  required  with  each  release  to  cross  reference 
from  the  developers  identification  system  to  the  Governments  system.  This  was  in  addition  to 
the  manual  coordination  required  after  each  change  control  board  when  CTA  would  update  the 
Government  database  and  then  have  to  perform  an  database  export  and  ship  the  database  to  the 
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developer  for  them  to  reload  into  their  system.  This  process  inevitably  led  to  discrepancies  in 
the  contents  between  the  two  databases  as  the  developers  implementation  was  also  evolving. 

Testing 

Testing  was  a  key  component  in  the  development  of  the  IV&V  program.  One  of  the  key 
concepts  of  the  EA/RP  process  is  the  “build-a-little,  test-a-little,  field-a-little”  philosophy.  This 
philosophy,  while  essential  to  developing  in  an  EA/RP  environment,  does  introduce  some  risk. 
The  methodology  chosen  to  mitigate  this  risk  from  the  IV&V  perspective  was  selective  in-depth 
testing  of  key  functional  threads  identified  as  part  of  the  “Spotlighting”  process. 

In  a  classical  acquisition,  the  IV&V  program  begins  with  a  System  Specification  from  which  a 
Requirements  Traceability  Matrix  is  built.  These  requirements  are  then  flowed  down  to  the 
Software  Requirements  Specification  (SRS)  and  Software  Design  Documents  (SDD);  then  these 
documents  are  used  by  the  IV&V  contractor  to  analyze  the  developed  product.  During  code  and 
implementation,  many  issues  arise  and  due  to  schedule  pressures  complete  documentation  rarely 
takes  place.  The  EA/RP  environment  tends  to  compound  the  deviation  between  the  "build  to" 
documentation  as  compared  to  the  "as  built"  system.  The  application  of  reverse  engineering 
tools  allowed  the  rapid  updating  of  the  "build  to"  documents,  thus  ensuring  that  the  specification 
represented  the  actual  code. 

To  reflect  the  reality  of  the  EA/RP  approach,  the  testing  philosophy  employed  consisted  of  two 
elements.  The  first  element  was  identifying  the  key  functional  threads  that  were  new  or 
significantly  modified  in  each  prototype  and  consequently  required  “spotlighting”  as  part  of  the 
engineering  test  process.  The  second  element  in  the  test  program  was  support  to  the  FQT 
Program  between  the  Government  and  the  developer. 

Spotlight  testing  was  designed  to  ensure  that  each  function  was  complete  enough  and  robust 
enough  to  ensure  a  valid  evaluation  by  the  users  during  the  user  evaluation  portion  of  the 
prototype  schedule.  This  testing  was  typically  very  intense  from  a  schedule  perspective  as  the 
candidate  code  for  the  prototype  was  typically  not  available  until  5  to  7  days  prior  to  the 
evaluation  session.  This  typically  allowed  a  maximum  of  3  days  for  the  spotlight  effort  to  test 
and  report  problems  so  that  a  correction  could  be  incorporated  by  the  developer  prior  to  the  first 
day  of  the  evaluation  session.  After  an  evaluation  the  functions  were  tested  from  an  engineering 
perspective  for  accuracy  and  stability. 

The  IV&V  testing  of  APS  was  based  upon  both  “black  box”  and  “white  box”  testing  concepts. 
The  white  box  testing  strategy  resulted  in  deriving  “operationally  based”  test  scenarios  to 
examine  the  functionality  of  the  system  from  a  real  world  user  perspective.  White  box  tests 
were  derived  for  crucial  functional  strings  where  it  was  imperative  to  determine  the  veracity  of 
the  algorithms  and  their  implementation. 

A  key  element  of  the  “black  box”  approach  to  validating  the  APS  software  was  the 
incorporation  of  "operational  test"  strategies  during  the  early  stages  of  APS  development. 
Operationally  based  test  scenarios  were  devised  early  in  the  IV&V  program  and  were  performed 
as  part  of  the  Functional  Prototype  tests.  These  test  scenarios  were  constructed  around  "real 
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world"  scenarios,  thereby  ensuring  that  realistic  data  in  terms  of  content  and  quantity  was 
employed.  This  data  included  realistic  forces,  threats,  system  parametrics  (aircraft,  missiles, 
radar,  radios,  etc.).  The  test  scenario  was  constructed  to  ensure  that  all  United  States  Message 
Text  Format  (USMTF)  mission  types  that  APS  was  required  to  support  would  be  constructed 
and  output  in  the  Air  Tasking  Order  (ATO )  so  that  the  content  and  format  of  *e  ATO  could  be 
validated.  The  scenario(s)  were  designed  as  an  end  to  end  test  (i.e.,  from  receipt  of  a  Target 
Nomination  List  (TNL)  to  the  ATO  output)  of  the  APS  software. 

A  major  issue  associated  with  this  operational  testing  was  assuring  that  the  system  was  stressed 
with  data  and  functional  loads  which  accurately  reflected  the  loads  that  could  be  expected  by  the 
fielded  version(s)  of  APS.  Examples  of  where  problems  were  identified  by  the  operational  test 
were  the  ACC  version  in  which  the  user  interface  supported  the  display  of  approximately  4000 
separate  entries  for  Standard  Configuration  Loads  (SCLs),  where  the  operational  requirement 
(and  actual  data  base)  consisted  of  over  5000  entries.  Attempting  to  retrieve  and  display  SCL 
data  with  a  data  set  of  this  size  resulted  in  the  APS  User  Interface  application  crashing,  thereby 
preventing  the  users  from  accomplishing  their  planning  task.  This  problem  was  discovered 
during  the  ACC  test  and  was  subsequently  corrected  in  the  next  formal  release.  This  example 
highlights  the  importance  of  testing  with  "real  world"  data  as  this  function  had  always  performed 
satisfactorily  during  DT&E  levels  of  testing  with  "test  data"  and  limited  quantities  populated  in 
the  data  base. 

In  order  to  assure  ATO  message  compliance  for  missions  which  are  hnked  such  as  the  set  of 
USMTF  (REFUEL,  7REFUEL,  and  EC),  the  operational  test  was  developed  so  that  realistic 
support  mission  requests/requirements  were  created  and  were  planned  from  the  appropriate  APS 
support  screen.  The  operational  test  also  checked  the  interrelationships  between  missions  as  they 
appeared  on  the  support  work  screens  assuring  that  as  missions  were  created  or  deleted  that  the 
support  screens  and  requirements  were  updated  properly.  In  particular  that  all  missions  that 
generated  support  requests  were  displayed  on  the  appropriate  (Tanker  ,  EC)  screen  and  also  if 
missions  were  deleted  that  had  support  requests  associated  with  them  that  the  resources  were 
returned  to  the  queue  to  be  reused. 


The  operational  test  was  exercised  as  an  end  to  end  test  from  the  input  of  a  TNL  to  the  output  of 
the  finished  ATO.  After  the  end  to  end  test  execution  the  individual  message  set  items  contained 
within  the  ATO  were  traced  to  their  origins  such  as  target  locations  in  the  TGTLOC  set  of  the 
ATO  and  were  in  fact  the  Desired  Mean  Point  of  Impact  (DMPI)  found  in  the  original  source 
TNL.  Each  mission  was  traced  to  assure  accuracy  of  both  pass  through  data  and  system 
generated  data  (call  signs  were  for  the  proper  day,  IFF/SIF  codes  were  not  duplicated  etc.) 

The  IV&V  test  approach  also  recognized  the  fact  that  to  check  the  veracity  of  the  code,  specific 
algorithms  needed  an  in-depth  examination  to  verify  that  the  implementation  was  correct  and 
reflected  truth  These  areas  were  tested  using  white  box  techniques.  Specific  threads  that  could 
be  identified  as  crucial  to  the  planning  process  were  identified. 

The  examination  of  the  refueling  models  consisted  of  capturing  the  design  for  a  release  into  the 
Cadre  Teamwork  environment  via  the  reverse  engineering  tool  or  by  analysis  of  the  code  and 
manually  entering  the  data.  The  analysis  tools  provided  by  Cadre  were  then  executed  to  check 


consistency  of  the  design  and  interfaces.  Specific  test  drivers  were  also  prepared  to  execute 
selected  modules.  Analysis  of  the  design  showed  that  the  basic  design  was  extremely  limited 
with  regards  to  the  refueling  scenarios  supported  and  required  additional  work  to  meet  “real 
world  operational  requirements.  A  number  of  discrepancies  between  the  various  code 
components  were  also  discovered  as  a  result  of  this  process.  The  most  serious  from  a  planning 
perspective  was  that  various  developers  implemented  their  portion  of  the  code  using  different 
units  of  measure  internally.  Therefore  while  data  could  be  passed  back  and  forth  between  the 
various  modules,  the  final  result  was  that  the  computations  for  determining  if  an  aircraft  required 
refueling,  how  long  it  would  take  to  perform  the  refueling,  and  the  amount  of  fuel  onboard  the 
tanker  were  not  in  agreement.  Among  the  various  assumption  in  use  were  that  fuel  was  being 
tracked  either  in  pounds,  hundreds  of  pounds,  or  thousands  of  pounds.  This  resulted  in  all 
missions  that  planned  refueling  to  have  incorrect  schedules  generated  for  the  amount  of  fuel 
required,  and  the  time  to  accomplish  a  refueling.  These  findings  were  documented  in  a  series  of 
reports  to  the  Government  and  as  SCRs  which  resulted  in  a  much  improved  set  of  refueling 
algorithms  by  the  time  the  program  reached  Phase  V. 


The  second  element  of  the  test  program  was  IV &V  support  to  the  formal  test  program  between 
the  Government  and  the  developer.  The  IV&V  contractor  and  the  Government  were  the 
responsible  agency  responsible  for  the  acceptance  of  the  final  product.  The  development  team 
was  not  provided  manpower  to  perform  this  testing.  Rather  than  have  two  testing  cycles  in 
series,  the  APS  program  office  decided  to  eliminate  the  contractor's  testing  at  the  system  level, 
and  to  ensure  a  quality  product,  assumed  the  responsibility  of  conducting  the  FQT.  This 
provided  an  independent  evaluation  of  the  release  and  afforded  the  program  office  a  hands  on 
feel  for  the  product. 

FQT  for  the  APS  program  also  consisted  of  a  number  of  other  differences  in  additon  to  the 
Government  being  the  responsible  agency.  Typically  FQT  is  conducted  at  the  developer’s  site, 
the  software  accepted  by  the  Government  and  then  installed  at  operational  sites  by  the 
Government  with  support  on  a  “as-needed”  basis  by  the  developer.  Reflecting  the  high  user 
involvement  in  the  APS  program  the  first  portion  of  the  FQT  was  conducted  at  the  developer’s 
site,  and  after  verifying  the  release,  the  FQT  team  (Government,  developer,  and  CTA)  traveled 
on-site  and  installed  the  system.  The  FQT  was  then  re-run  as  part  of  the  installation  and  training 
activity  to  ensure  that  the  user’s  requirements  were  met  and  that  the  system  operated  in  the  real 
world  conditions  present  at  the  operational  sites.  These  additional  tests  were  used  to  validate  the 
FQT  tests  and  scenarios.  Some  errors  were  encountered  during  these  additional  FQT  tests.  The 
majority  of  these  errors  were  a  result  of  the  different  hardware  and  software  configurations 
which  existed  at  the  operational  sites  and  which  could  not  have  been  replicated  at  either  the 
developer’s  facility  or  the  APS  Integration  Lab  at  Rome  Laboratory. 

As  part  of  the  IV&V  responsibility  for  FQT  support,  CTA  developed  and  supplied  the  test 
scenario  and  accompanying  database  used  to  support  the  qualification  test  activities.  This  effort 
was  initiated  by  CTA  and  completed  in  conjunction  with  the  Government  and  the  developer. 

This  database  then  served  as  the  baseline  for  all  subsequent  formal  test  activities  on  the  APS 
program.  This  database  was  built  based  upon  the  real  world  conditions  experienced  during 
Desert  Storm.  This  was  an  important  step  in  convincing  the  user  to  accept  APS,  in  that  the 
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The  requirement  to  support  the  users'  schedule  did  not  allow  the  accepted  process  and  time  line 
associated  with  a  classical  software  acquisition  for  the  conduct  of  the  audits.  Not  only  was 
schedule  time  a  concern,  but  the  resources  to  perform  a  manual  audit  of  the  code  were  not 
available.  This  problem  was  recognized  early  in  the  program  and  was  a  key  element  when  the 
assessment  to  choose  a  modern  SEE  was  performed  and  the  assessment  of  configuration 
management  practices  was  performed.  Part  of  the  evaluation  criteria  for  both  the  SEE  and  CM 
required  that  the  products  and  procedures  selected  had  to  support  capturing  the  required  data  and 
automating  the  audit  process  to  support  the  installation  schedule  required  to  support  the  user 
community. 

To  support  the  schedule  requirements,  particularly  in  the  PCA,  which  is  the  most  resource 
consuming,  the  process  of  the  physical  code  inspection  was  automated  using  the  tools  provided 
as  part  of  the  SEE.  This  process  worked  particularly  well  with  the  ability  to  display 
requirements,  design,  and  code  on  a  screen  together  to  compare  then  against  each  other.  The 
SEE  also  provided  tools  to  perform  audits  on  the  code  for  items  such  as  variables  declared  but 
never  used  or  not  initialized. 

Data  Validation/Integritv 

In  discussing  APS,  the  concept  of  veracity  and  the  issues  of  data  validity  and  database  design  are 
inexorably  linked.  This  linkage  occurred  as  a  result  of  the  development  methodology  and  the 
developers  attempt  to  implement  functions  prior  to  fully  understanding  the  area  and  having  a 
complete  set  of  requirements  defined.  It  therefore  became  crucial  that  the  underlying  data  model 
be  complete,  flexible  and  reflect  the  needs  of  the  algorithms  supported.  There  were  two 
important  considerations  in  this  area,  the  first  was  that  the  data  representation  internally  and  the 
final  ATO  product  had  to  be  compliant  with  the  USMTF  standards,  and  the  second  factor  was 
the  design  of  the  database  and  the  potential  for  problems  both  with  data  integrity  and  data  base 
performance  adversely  impacting  overall  system  performance. 

The  area  that  represented  the  highest  risk  to  the  APS  program  from  the  IV&V  perspective  was 
the  methodology  and  design  of  the  database  that  was  being  employed  by  the  development 
contractor  when  CTA  came  on  board.  The  approach  in  use  by  the  developer  was  to  start  with  a 
simple  model  and  to  continually  add  tables  and/or  define  new  relationships  as  the  user  pointed 
out  problems/issues  at  the  evaluation  sessions.  While  this  approach  is  adequate  in  a  small 
system,  when  the  volume  of  data  that  is  required  in  APS,  and  the  complexity  of  the  relationships 
required  to  define  the  Air  Battle  Plan  (ABP)  are  considered,  this  methodology  poses  significant 
risks  to  the  Government.  In  particular  it  would  result  in  a  system  with  the  same  data 
represented  in  multiple  places,  and  lead  to  a  corresponding  increase  in  complexity  of  the  queries 
required  to  keep  all  the  data  sets  synchronized.  This  ultimately  would  impact  the  system  in  two 
areas;  the  ability  of  the  Air  Force  to  maintain  the  system  would  be  curtailed  and  system 
performance  would  be  degraded. 

While  the  developer  team  had  personnel  well  versed  in  data  base  design  methodology  there  was 
little  or  no  domain  expertise  available  to  the  database  designers  to  ensure  that  the  model  being 
developed  was  complete  or  accurately  reflected  the  true  relationships  required  to  capture  the  data 
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required  for  the  ABP,  and  then  convert  it  into  the  format  in  which  it  is  disseminated,  namely  the 
Air  Tasking  Order  (ATO). 

With  the  concurrence  of  the  Government,  CTA  provided  an  extensive  review  and  design 
suggestions  for  portions  of  the  data  model  in  those  areas  where  CTA  staff  possessed  extensive 
domain  expertise.  In  particular,  the  targeting  model,  the  refueling  model,  and  the  packaging 
model  were  singled  out  via  the  spotlight  methodology  as  areas  that  were  of  high  risk  and 
required  extensive  investigation.  The  specific  criteria  was  that  these  models  were  the  most 
crucial  to  the  success  of  the  system  and  were  also  the  most  complicated  in  terms  of  the  data 
required  to  define  the  model  as  well  as  the  relationships  that  had  to  be  defined  and  understood  by 
the  developer  between  the  data  within  each  model  and  the  relationships  between  the  individual 
models. 

The  second  area  of  concern  throughout  the  APS  program  has  been  the  adequate  specification  of 
external  data  sources  to  populate  the  APS  database  to  support  both  the  initial  installation  in  a 
specific  theater  of  operations  and  subsequently  appropriate  sources  for  the  daily  situation  updates 
required  to  support  the  daily  planning  cycle. 

The  developer  assumed  that  they  would  be  provided  electronic  sources  to  interface  with  and 
initially  made  little  provision  for  allowing  setup  data  to  be  input  manually.  The  key  element 
here  in  the  opinion  of  CTA  was  the  lack  of  domain  expertise  possessed  by  the  development 
team.  The  developers  understood  the  volume  of  data  that  APS  would  require  prior  to  initiating  a 
planning  cycle  but  could  not  appreciate  the  fact  that  electronic  feeds  were  not  available  or  that 
significant  portions  of  the  data  did  not  even  exist  electronically.  It  took  repeated  design  reviews 
and  meetings  to  convince  the  developer  that  the  majority  of  the  data  they  were  requesting  either 
did  not  exist  in  electronic  format  or  would  not  be  available  to  the  AOC  in  the  format  that  APS 
required.  As  a  result  the  initial  design  to  support  data  input  to  APS  was  not  designed  at  the  same 
time  as  the  rest  of  the  database.  This  led  to  a  number  of  discrepancies  such  as  user  supplied  data 
not  being  stored  in  the  data  base  for  later  use  and  significant  discrepancies  between  what  the 
User  Interface  allowed  as  legal  input  (format,  length,  restricted  characters),  what  the  data  base 
allowed,  and  what  the  standards  (i.e.  USMTF )  specified  as  legal  formats.  This  analysis  was  an 
ongoing  task  through  the  M_AB  baselines  as  the  developer  continually  refined  the  process  based 
upon  rV&V  testing  results  and  feedback  from  the  users.  In  the  current  version  of  APS  (C_AD), 
the  majority  of  the  problems  associated  with  the  initial  load  of  APS  have  been  corrected.  The 
User  Interface,  the  database,  and  the  import  function  (for  those  sources  which  are  available)  are 
all  consistent  in  their  processing  and  use  the  USMTF  as  the  definition  for  legal  field  lengths  and 
character  sets.  For  those  installations  that  still  operate  APS  in  a  standalone  manner  the  data 
input  requirements  can  still  represent  an  onerous  burden  for  the  system  administration  staff  or 
whomever  is  tasked  to  maintain  the  data  base.  In  a  dynamic  environment  where  significant 
changes  to  a  TNL  were  required  daily  ( i.e.  a  500+  target  scenario  with  weaponeering)  the 
current  setup  screens  and  process  would  limit  the  effectiveness  of  APS  due  to  the  time  and 
manpower  requirements  to  input  the  data  prior  to  the  start  of  the  planning  session. 
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CONCLUSION 


The  APS  rv&V  effort  represented  an  attempt  to  explore  alternative  concepts  and  methodologies 
in  performance  of  an  IV&V  effort  in  an  EA/RP  environment.  Classical  IV&V  techniques  were 
deemed  to  be  unresponsive  to  the  dynamic  nature  and  continually  evolving  baseline  that  is 
characteristic  of  the  EA/RP  environment.  Alternative  concepts  and  methodologies  were 
examined  and  tailored  to  be  responsive  to  the  unique  environment  and  challenges  represented 
by  a  software  development  effort  accomplished  using  an  EA/RP  methodology. 

The  APS  IV&V  effort  specifically  pursued  the  concepts  of  “Spotlighting”,  “Software  Veracity”, 
and  extensive  usage  of  a  modern  Software  Engineering  Environment  to  aid  both  the  software 
designers  and  the  IV&V  engineers.  The  adoption  of  "Spotlighting"  allowed  IV&V  personnel  to 
focus  on  a  specific  area  when  it  was  crucial  to  the  program  and  then  essentially  put  that  activity 
into  a  "maintenance"  mode  once  the  area  had  been  adequately  addressed.  Software  veracity  was 
a  technique  established  to  ensure  that  the  individual  functional  requirements  were  implemented 
completely  and  interacted  with  the  rest  of  the  system  properly.  A  SEE  was  recommended  and 
adopted  by  the  program  both  to  encourage  modern  software  engineering  practices  and  to  allow  a 
rapid  analysis  capability  of  design  components  as  part  of  the  verification  process  and  to  audit 
and  accept  each  new  delivered  baseline. 

The  core  ingredients  to  the  APS  IV&V  effort  being  a  positive  factor  towards  the  timely  delivery 
of  a  quality  APS  product  in  the  EA/RP  environment  was  a  combination  of  two  factors:  the 
cooperative  working  arrangement  that  evolved  between  the  Air  Force  Program  Office,  the 
developer,  and  the  IV&V  contractor,  and  the  development  of  an  IV&V  methodology  that 
allowed  the  IV&V  contractor  to  continually  leverage  the  limited  resources  available. 
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Mission.  The  mission  of  Rome  Laboratory  is  to  advance  the  science  and 
technologies  of  command,  control,  communications  and  intelligence  and  to 
transition  them  into  systems  to  meet  customer  needs.  To  achieve  this, 
Rome  Lab: 


a.  Conducts  vigorous  research,  development  and  test  programs  in  all 
applicable  technologies; 

b.  Transitions  technology  to  current  and  future  systems  to  improve 
operational  capability,  readiness,  and  supportability; 

c.  Provides  a  full  range  of  technical  support  to  Air  Force  Materiel 
Command  product  centers  and  other  Air  Force  organizations; 

d.  Promotes  transfer  of  technology  to  the  private  sector; 

e.  Maintains  leading  edge  technological  expertise  in  the  areas  of 
surveillance,  communications,  command  and  control,  intelligence,  reliability 
science,  electro-magnetic  technology,  photonics,  signal  processing,  and 
computational  science. 


The  thrust  areas  of  technical  competence  include;  Surveillance, 
Communications,  Command  and  Control,  Intelligence,  Signal  Processing, 
Computer  Science  and  Technology,  Electromagnetic  Technology, 
Photonics  and  Reliability  Sciences. 


